feat(CI): Keep local and Github CI test in sync with `cargo xtask` by 2010YOUY01 · Pull Request #23414 · apache/datafusion

2010YOUY01 · 2026-07-09T08:41:29Z

Summary

For easier review, start with this PR writeup, then read the top comments in .github/workflows/rust.yml, and then follow along with the code.

This PR introduces an easy way to reproduce CI checks locally.

To reproduce .github/workflows/rust.yml:

cargo xtask ci workflow rust

Example output:

yongting@Yongtings-MacBook-Pro-2 ~/C/datafusion (ci-xtask)> cargo xtask ci workflow rust

ci workflow `rust` total execution time: 565.131385s
    ci job `linux-rustdoc`: 210.992471s
    ci job `linux-test-doc`: 117.395768s
    ci job `linux-test`: 85.890313s
    ci job `linux-test-example`: 39.367084s
    ci job `linux-test-datafusion-cli`: 37.235765s
    ci job `clippy`: 24.750692s
    ci job `msrv`: 16.421963s
    ci job `macos-aarch64`: 12.399443s
    ci job `config-docs-check`: 6.801263s
    ci job `check-fmt`: 3.139787s
    ci job `linux-cargo-check-datafusion`: 2.715153s
    ci job `cargo-toml-formatting-checks`: 1.94445s
    ci job `examples-docs-check`: 1.518475s
    ci job `linux-cargo-check-datafusion-functions`: 1.121708s
    ci job `sqllogictest-substrait`: 991.08ms
    ci job `linux-build-lib`: 600.417ms
    ci job `linux-datafusion-proto-features`: 596.001ms
    ci job `linux-datafusion-substrait-features`: 528.707ms
    ci job `vendor`: 346.94ms
    ci job `linux-datafusion-common-features`: 208.197ms
    ci job `verify-clean`: 83.283ms
    ci job `verify-clean`: 82.425ms

To reproduce single job or step similarly,

yongting@Yongtings-MacBook-Pro-2 ~/C/datafusion (ci-xtask)> cargo xtask ci job check-fmt
+ /Users/yongting/Code/datafusion/ci/scripts/rust_fmt.sh
[rust_fmt.sh] `cargo fmt --all -- --check`

ci job `check-fmt` total execution time: 2.817591s
    ci step `fmt`: 2.817591s
yongting@Yongtings-MacBook-Pro-2 ~/C/datafusion (ci-xtask)> cargo xtask ci step fmt
+ /Users/yongting/Code/datafusion/ci/scripts/rust_fmt.sh
[rust_fmt.sh] `cargo fmt --all -- --check`

ci step `fmt` completed in 2.793302s

Background

The existing GitHub CI follows a 3-layer structure:

step: an atomic check, for example cargo fmt
job: one or more steps, for example check-fmt
workflow: a list of jobs, for example .github/workflows/rust.yml

For example:

workflow: rust
  job: check-fmt
    step: fmt
      command: cargo fmt --all -- --check

  job: clippy
    step: clippy
      command: cargo clippy ...

  job: linux-test
    step: rust-test
      command: cargo test ...
    step: verify-clean
      command: git diff --exit-code

This PR mirrors that structure in xtask, so the same CI units can be used by both GitHub Actions and local runs.

Motivation

Making CI easy to reproduce locally improves developer experience.

If a specific job fails on GitHub, we can reproduce it locally with cargo xtask ci job <job>.
After a small change, we can run a subset of the full CI locally. This shortens the development cycle while still giving good confidence that the full CI will pass.

Possible future extensions:

cargo xtask ci workflow changed

Detect changed Rust files, then only run unit tests and clippy on crates affected by the change, plus relevant integration tests.

cargo xtask ci workflow doc

For documentation-only changes, only run formatting, typo checks, docs checks, and other lightweight checks.

Implementation

The implementation adds reusable units for atomic checks, such as cargo fmt. In GitHub Actions terminology, these are "steps".

The key idea is:

GitHub CI orchestrates these atomic checks remotely.
cargo xtask ci ... orchestrates the same checks locally.

There are two viable ways to organize these atomic checks:

cargo xtask, for example cargo xtask ci step fmt
shell scripts, for example ./check_fmt.sh

I previously tried the shell script approach for dev.yml:

https://github.com/apache/datafusion/blob/main/dev/rust_lint.sh

For the current CI complexity, I found cargo xtask easier to keep organized. I suggest we continue with this approach and later migrate the existing scripts to use xtask as well.

Scope of this PR

This PR:

Introduces the xtask framework.
Migrates the steps in rust.yml to xtask runners. The commands are unchanged, except for the small difference noted below.
Supports local orchestration with cargo xtask ci workflow rust.

Notes:

This PR only achieves CI check command equivalence. Setup steps and tool installation are not fully reproduced yet, and are expected to be done manually for now.
Three jobs are skipped in the local run because they need slightly different handling on macOS. I would like to add them in follow-up PRs to keep this PR easier to review.

Testing

Individual 'step's will be tested by CI.

For workflow/jobs, I have tested locally on MacOS. I think we can let CI to test some fast workflows in the future.

2010YOUY01 · 2026-07-09T08:44:04Z

+                "--release-source",
+                "rust-dist",


These two lines added should be the only changed command (I found the default option fails on my local setting)

2010YOUY01 · 2026-07-09T09:02:27Z

cc who might be interested, @alamb @comphead @kumarUjjawal @blaginin

comphead

Thanks @2010YOUY01 does that mean CI changes should be reflected in xtask and vice versa?

WDYT about https://github.com/nektos/act to try run DataFusion CI locally, although I heard not all things possible with this tool

kumarUjjawal · 2026-07-09T15:04:39Z

+        .iter()
+        .find(|job| job.name == VERIFY_CLEAN_JOB_NAME)
+        .unwrap_or_else(|| panic!("missing `{VERIFY_CLEAN_JOB_NAME}` job"));
+    let mut jobs = vec![verify_clean];


fails immediately if we have there are any uncommitted changes before it runs any CI check. Was this intentional?

Yes, this is intentional, now the whole workflow requires a clean local git repo to run (no uncommitted changes), though I think there is room to improve it later.

To better simulate GitHub CI, we should try to reproduce the actual commands it runs as closely as possible.

Currently, GitHub CI runs several verify-clean checks between tests to ensure that no generated or modified files are left behind after each test, using git diff.

So for the local reproducer to behave the same way, we need to ensure there are no uncommitted changes before running the full workflow.

kumarUjjawal · 2026-07-09T15:07:59Z

+            format!("failed to create {}: {error}", tpch_data.display())
+        })?;
+
+        self.command("git")


if the step fails partway through and I re-run, git clone immediately fails with "destination path already exists". I'd have to manually rm -rf tpch-dbgen before retrying.

It keeps the existing command, we could fix it later. See #23414 (comment)

kumarUjjawal · 2026-07-09T15:09:48Z

+            .env("RUSTFLAGS", rustflags)
+            .run()?;
+
+        let chrome_driver = format!(


This results in wasm-pack being called with --chromedriver /chromedriver , which fails with a confusing "not found" error. The run_rust_sqllogictest function handles the missing-env-var case much better by returning an explicit error, we can use the same here

Similarly, #23414 (comment)

I found this job also a bit tricky to setup locally on my Mac

kumarUjjawal · 2026-07-09T15:15:14Z

+            let result = runner(self, &args);
+            job_timing.add_step_timing(step, start.elapsed().unwrap_or_default());
+
+            if let Err(error) = result


we should consider building the full message in a single map_err rather than chaining, there are several instances

2010YOUY01 · 2026-07-10T01:14:04Z

Thanks @2010YOUY01 does that mean CI changes should be reflected in xtask and vice versa?

Yes, exactly. That's the end goal, and this PR is the initial step with a limited scope.

WDYT about https://github.com/nektos/act to try run DataFusion CI locally, although I heard not all things possible with this tool

Good to know about act! I went to do some investigation, its limitations are

Not mature enough (lots of unsupported features for github .yml, bugs)
Expensive to run locally (Docker based, pulls large images)

Overall I think xtask is a better choice.

2010YOUY01 · 2026-07-10T01:20:59Z

+use crate::ci_workflows::WorkflowInfo;
+
+const VERIFY_CLEAN_JOB_NAME: &str = "verify-clean";
+const SKIPPED_JOBS: &[&str] = &[


These jobs are not supported by the local runner yet. I found that they need some modifications to run on macOS.

This PR does not change any commands, to make the review easier.

alamb

Thank you @2010YOUY01 for this PR -- the idea of making the deveopment experience better is good, but I am worried about adding another tool requirement

Making CI easy to reproduce locally improves developer experience.

I do agree with this, but I also think making the development experience more bespoke (requiring yet another tool -- we already have taplo, prettier, etc) works against this purpose

For example InfluxData uses just for this purpose, and I have also used Make for this purpose. This has resulted in several times it not being possible to use standard cargo commands to run the tests (b/c the indirection layers add some magic flag)

In my opinion these tools all tend to make understanding how to do things in the repo less standard and thus put up a larger barrier to entry.

On the other hand AI tools lower this barrier, so maybe it is not as much of a problem anymore 🤔

I previously tried the shell script approach for dev.yml:
https://github.com/apache/datafusion/blob/main/dev/rust_lint.sh

For the current CI complexity, I found cargo xtask easier to keep organized. I suggest we continue with this approach and later migrate the existing scripts to use xtask as well.

I agree we can't mirror CI 1:1 in shell scripts without unwarranted complexity

However I think we can get like 95% of the common failures from CI via shell scripts and that would be good enough for the repo

alamb · 2026-07-10T17:53:46Z

+# Architecture
+# -------------
+#
+# Each atomic CI check lives in an individual `xtask` step. The GitHub workflow


I have never heard of xtask -- I am sure it is awesome, but it does now add one more development dependency for developers and makes it that much harder (maybe just a little) to develop in DataFuson. (as now you need to have yet another tool installed)

alamb · 2026-07-10T17:54:02Z

+#
+# (@ means no `needs` dependency.)
+#
+# @ -> linux-build-lib


Is the idea that xtask can model this type of dependency tree?

alamb · 2026-07-10T18:00:16Z

So in other words maybe we don't have to have 100% parity with what CI does -- instead we can focus on getting the most of the coverage via standard commands.

And maybe if there is something that CI is doing that is non standard, we can try to reduce the complexity there

feat(CI): Keep local and Github CI test in sync with cargo xtask

17d62e7

github-actions Bot added the development-process Related to development process of DataFusion label Jul 9, 2026

2010YOUY01 commented Jul 9, 2026

View reviewed changes

2010YOUY01 mentioned this pull request Jul 9, 2026

EPIC: Add Local Scripts to Reproduce Full CI #21048

Open

12 tasks

comphead reviewed Jul 9, 2026

View reviewed changes

kumarUjjawal reviewed Jul 9, 2026

View reviewed changes

2010YOUY01 commented Jul 10, 2026

View reviewed changes

alamb reviewed Jul 10, 2026

View reviewed changes

Uh oh!

Conversation

2010YOUY01 commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Background

Motivation

Implementation

Scope of this PR

Testing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 commented Jul 9, 2026

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

2010YOUY01 commented Jul 9, 2026 •

edited

Loading

2010YOUY01 commented Jul 10, 2026 •

edited

Loading